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determine  its  value.  We  take  seriously  the  pernicious  effects  of  the  so-called  “theory- 
practice”  gap,  which  would  separate  the  acquisition  scholar  from  the  acquisition  practitioner, 
and  relegate  the  scholar’s  work  to  mere  academic  “shelfware.”  Some  design  features  of  our 
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project  findings  as  a  condition  of  funding  award;  “pushing”  potentially  high-impact  research 
reports  (e.g.,  via  overnight  shipping)  to  selected  practitioners  and  policy-makers;  and  most 
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Panel  16  -Contributions  of  Systems  Engineering  to 
Effective  Acquisition 


Thursday,  May  12,  2011 


9:30  a.m.  -  Chair:  Rear  Admiral  John  Clarke  Orzalli,  USN,  Vice  Commander,  Naval  Sea 
1 1 :00  a.m.  Systems  Command 

Control  of  Total  Ownership  Costs  of  DoD  Acquisition  Development 
Programs  Through  Integrated  Systems  Engineering  Processes  and 
Metrics 

Paul  Montgomery  and  Ron  Carlson,  NPS 

Applying  an  Influencer  Approach  to  Ingrain  Systems  Engineering  into  Pre- 
Milestone  B  Defense  Programs 

Bob  Keane,  Ship  Design  USA,  Inc. 

Factors  Influencing  the  Effectiveness  of  Systems  Engineering  Training 
and  Education  in  the  Department  of  Defense 

William  Fast,  NPS 


Rear  Admiral  Orzalli — Vice  Commander,  Naval  Sea  Systems  Command  (NAVSEA).  Rear  Admiral 
Orzalli  is  the  son  of  a  retired  Navy  captain.  He  graduated  with  distinction  from  the  U.S.  Naval 
Academy  in  1978. 

At  sea,  he  served  aboard  USS  Snook  (SSN  592)  as  an  engineering  division  and  weapons  officer;  and 
as  USS  Helena’s  (SSN  725)  engineering  officer.  Ashore,  Orzalli  has  served  at  the  U.S.  Naval 
Academy,  as  well  as  tours  at  naval  shipyards  in  Mare  Island,  Puget  Sound,  and  Portsmouth. 

Orzalli  was  the  45th  shipyard  commander  at  Puget  Sound  Naval  Shipyard  from  2002-2005.  During 
his  command  tour,  he  assumed  additional  duties  in  establishing  the  Northwest  Regional  Maintenance 
Center.  Following  selection  to  flag  rank,  Orzalli  was  the  deputy  director,  Fleet  Readiness  Division, 
OPNAV  (N43B);  commanding  officer,  Mid-Atlantic  Regional  Maintenance  Center,  then  established 
commander,  Regional  Maintenance  Centers. 

Most  recently,  Orazalli  was  the  director,  Fleet  Maintenance  on  the  staff  of  commander,  U.S.  Fleet 
Forces  Command.  His  service  decorations  include  the  Legion  of  Merit  (with  four  stars),  the 
Meritorious  Service  Medal  (with  two  stars),  Navy  Commendation  Medal  (with  star),  Navy  and  Marine 
Corps  Achievement  Medal  (with  three  stars)  and  various  other  unit  and  operational  awards. 

Orzalli  holds  a  Bachelor  of  Science  in  Marine  Engineering  from  the  U.S.  Naval  Academy,  Naval 
Engineer,  a  Master  of  Materials  Science  and  Engineering  from  Massachusetts  Institute  of 
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Factors  Influencing  the  Effectiveness  of  Systems  Engineering 
Training  and  Education  in  the  Department  of  Defense 

William  Fast — COL,  USA  (Ret.).  COL  Fast  facilitates  acquisition  and  program  management  courses 
at  the  Naval  Postgraduate  School.  He  also  writes  and  speaks  on  various  management  topics  and 
provides  consultation  services  to  defense  acquisition  programs.  From  2006-2010,  COL  Fast  taught 
program  and  financial  management  courses  at  the  Defense  Acquisition  University. 

Abstract 

While  current  systems  engineering  certification  courses  within  the  Department  of 
Defense  appear  to  do  a  pretty  good  job  of  training  and  educating  the  workforce, 
improvements  can  be  made.  The  use  of  more  problem-based  methods  of  learning 
would  equip  the  students  with  better  problem  identification  and  reasoning  skills 
needed  to  solve  the  complex  problems  they  encounter  on  the  job.  Learning 
outcomes  in  some  of  these  courses  could  be  rewritten  to  target  the  analyze, 
evaluate,  and  create  levels  of  Bloom’s  Taxonomy,  thereby  improving  student  critical 
thinking  skills  and  ultimately  improving  far-transfer  of  learning  to  the  job.  Also, 
learning  assessment  methods  in  a  few  of  the  courses  could  be  changed  to  focus 
more  on  the  assessment  of  conceptual  understanding,  vice  rote  memorization,  in 
order  to  promote  deep  learning.  Recommendations  are  also  presented  for  additional 
research  into  a  more  effective  systems  engineering  andragogy. 

Purpose 

Competency-based  training  for  defense  acquisition  workers  in  the  systems 
engineering  discipline  is  accomplished  through  a  continuum  of  four  courses  developed  and 
delivered  by  the  Defense  Acquisition  University  (DAU): 

■  SYS  101  Fundamentals  of  Systems  Engineering;  computer-based  distanced 
learning;  35  hours. 

■  SYS  202  Intermediate  Systems  Planning,  Research,  Development  and 
Engineering,  Part  1;  computer-based  distance  learning;  30  hours. 

■  SYS  203  Intermediate  Systems  Planning,  Research,  Development  and 
Engineering,  Part  2;  resident  course;  36  hours. 

■  SYS  302  Technical  Leadership  in  Systems  Engineering;  a  resident  course; 

68  hours. 

The  primary  purpose  of  this  research  was  to  determine  if  the  methods  and  objectives 
of  these  systems  engineering  certification  courses  encourage  a  deep  approach  to  learning 
and  far-transfer  of  that  learning  (i.e.,  the  students  are  able  to  apply  what  they  have  learned 
on  the  job).  Ultimately,  the  effectiveness  of  systems  engineering  training  within  the 
Department  of  Defense  does  affect  the  outcome  of  systems  acquisition  programs. 

Method 

A  literature  search  revealed  that  a  problem-based  approach  to  teaching  systems 
engineering,  with  the  primary  objective  of  developing  the  student’s  ability  to  reason  and 
solve  complex  problems  (i.e.,  develop  critical  thinking  skills),  would  result  in  deep  learning 
and  promote  skill  transfer  to  the  job.  Therefore,  I  decided  to  study  the  current  design  of 
systems  engineering  courses  to  determine  how  much  problem-based  instruction  was 
actually  used.  To  gauge  the  stimulation  of  student  critical  thinking  skills  in  these  courses,  I 
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examined  student  learning  outcomes  for  each  of  the  systems  engineering  courses 
discussed  above.  I  also  examined  the  learning  assessments  for  each  course  to  determine  if 
they  had  been  designed  to  promote  deep  or  surface  learning  by  the  students. 

Student  course  materials  and  learning  objectives  from  the  four  systems  engineering 
certification  courses  discussed  previously  were  analyzed  in  three  ways.  First,  student 
materials  were  inspected  to  determine  the  time  allocated  to  computer-based  training  (CBT), 
lectures,  and  problem-based  exercises  in  each  of  the  courses.  These  student  course 
materials  are  available  to  the  public  on  the  DAU  iCatalog  website  (DAU,  201 1 ). 

Second,  the  lesson  objectives  (expected  student  outcomes)  for  each  of  the  four 
systems  engineering  certification  courses  were  categorized  according  to  their  required 
levels  of  functional  thought,  using  Bloom’s  Taxonomy  (Bloom,  1956,  updated  in  accordance 
with  Figure  1).  Specifically,  the  measureable  action  verb  from  each  learning  objective  was 
placed  into  one  of  the  six  Bloom  categories  representing  the  cognitive  activity  required  by 
the  student  to  successfully  demonstrate  that  objective.  The  lesson  objectives  are  available 
to  the  public  on  the  DAU  iCatalog  website  (DAU,  2011). 


Figure  1.  Bloom’s  Taxonomy 

(Rodgers,  2011) 

Note.  This  figure  contrasts  the  original  (old)  version  with  an  updated  (new)  version;  note 
changes  in  top  two  levels. 

Third,  the  course  learning  objectives  (expected  student  outcomes)  from  three 
selected  systems  engineering  courses  developed  and  delivered  by  the  Naval  Postgraduate 
School  (NPS)  were  categorized  according  to  their  required  levels  of  functional  thought, 
using  Bloom’s  Taxonomy.  The  three  courses  were  as  follows: 

■  SE3100  Fundamentals  of  Systems  Engineering;  resident  and  distance 
learning;  5  quarter  hours  (3  lecture/2  lab);  equivalent  to  DAU  SYS  101,  SYS 
202,  and  SYS  203. 

■  SI3400  Fundamentals  of  Engineering  Project  Management;  distance 
learning;  5  quarter  hours  (3  lecture/2  lab);  equivalent  to  DAU  SYS  302. 

■  SE4012  Management  of  Advanced  Systems  Engineering;  distance  learning; 
4  quarter  hours  (4  lecture/0  lab);  equivalent  to  DAU  SYS  302. 
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The  course  learning  objectives  were  obtained  from  Professor  Gary  Langford  of  the 
Systems  Engineering  Department,  Graduate  School  of  Engineering  and  Applied  Sciences, 


NPS. 


Finally,  student  learning  assessments  were  examined  in  the  four  DAU  and  the  three 
NPS  systems  engineering  course  to  determine  if  the  current  assessments  promoted  deep  or 
surface  learning.  Surface  learning  is  promoted  by  assessments  that  emphasize  recall 
based  upon  rote  memorization.  Deep  learning  is  promoted  by  assessing  the  student’s 
understanding  of  topics  (Felder  &  Brent,  2005,  p.  64). 

Results 

Time  allocated  to  computer-based  training  (CBT),  lectures,  and  problem-based 
instruction  in  each  of  the  four  systems  engineering  certification  courses  developed  and 
delivered  by  DAU  are  found  in  Table  1. 


Table  1.  DAU  Systems  Engineering  Course  Hours  Categorized  by  Method  of 

Instruction 

(DAU,  2011) 


Course 

CBT  Hours 

Lecture  Hours 

Problem-Based  Hours 

Total  Hours 

SYS  101 

35 

35 

SYS  202 

30 

30 

SYS  203 

9 

27 

36 

SYS  302 

21 

47 

68 

Totals 

65 

30 

74 

169 

Percentage 

38.46% 

17.75% 

43.79% 

100% 

Lesson  learning  objectives  (expected  student  outcomes),  categorized  by  Bloom’s 
level,  for  each  of  the  four  systems  engineering  certification  courses  developed  and  delivered 
by  DAU  are  found  in  Table  2. 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  FOR  INFORMED  CHANGE  -  81  - 


Table  2.  DAU  Systems  Engineering  Course  Objectives  Categorized  by  Bloom’s 

Level 

(DAU,  2011) 


Course 

Remember 

Understand 

Apply 

Analyze 

Evaluate 

Create  Total  Objectives 

SYS  101 

14 

138 

1 

1 

154 

SYS  202 

1 

29 

30 

SYS  203 

3 

9 

5 

6 

12 

1 

36 

SYS  302 

2 

49 

14 

8 

24 

28 

125 

Totals 

20 

225 

20 

15 

36 

29 

345 

Percentage 

5.80% 

65.22% 

5.80% 

4.45% 

10.43% 

8.41% 

100% 

Course  learning  objectives  (expected  student  outcomes),  categorized  by  Bloom’s 
level,  for  three  selected  systems  engineering  courses  developed  and  delivered  by  NPS  are 
found  in  Table  3. 

Table  3.  NPS  System  Engineering  Course  Objectives  Categorized  by  Bloom’s 

Level 

(NPS,  2011) 


Course 

Remember 

Understand 

Apply 

Analyze 

Evaluate 

Create 

Total  Objectives 

SE  3100 

2 

1 

2 

2 

1 

2 

10 

SI  3400 

1 

1 

6 

1 

1 

10 

SE  4012 

2 

2 

1 

1 

1 

7 

Totals 

5 

4 

9 

4 

1 

4 

27 

Percentage 

18.52% 

14.81% 

33.33% 

14.81% 

3.70% 

14.81% 

100% 

The  types  of  assessments  used  in  the  four  DAU  systems  engineering  certification 
courses  and  the  three  NPS  systems  engineering  courses  are  found  in  Table  4. 
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Table  4. 


Types  of  Learning  Assessments  Used  in  DAU  and  NPS  Systems 
Engineering  Courses 

(DAU,  2011;  NPS,  2011) 


Assessment 

SYS  101  SYS  202  SYS  203 

SYS  302 

SE  3100 

SI  3400 

SE  4012 

Objective  Exam/Quiz 

XXX 

X 

X 

X 

X 

Subjective  Exam/Quiz 

X 

X 

X 

Homework 

X 

X 

X 

Discussion  Participation 

X 

X 

X 

X 

Reflective  Writing 

X 

Individual  Briefing 

X 

X 

X 

X 

X 

Individual  Project 

X 

X 

X 

Team  Project 

X 

X 

X 

X 

X 

Discussion 

The  results  of  this  research,  as  presented  in  Table  1,  reveal  that  DAU  systems 
engineering  certification  courses  use  a  mix  of  computer-based  training,  lecture,  and 
problem-based  instruction.  The  SYS  101  and  SYS  202  courses  are  designed  as  computer- 
based  training  for  individuals  and  have  no  student-led  problem-solving  exercises.  However, 
SYS  202  does  use  an  integrated  case  study  to  help  the  student  understand  systems 
engineering  in  the  context  of  a  notional  defense  weapon  system.  The  SYS  203  course  was 
designed  as  the  exercise  extension  of  SYS  202,  so  students  in  SYS  203  spend  about  75% 
of  the  class  time  in  problem-solving  exercises.  Problem-solving  exercises  also  account  for 
69%  of  the  class  hours  in  the  SYS  302  course.  Thus,  two  of  the  four  DAU  systems 
engineering  certification  courses  do  provide  students  with  significant  amounts  of  problem- 
based  instruction. 

With  respect  to  student  learning  objectives,  the  DAU  SYS  203  and  SYS  302  courses 
grade  out  at  higher  Bloom’s  levels  ( analyze ,  evaluate,  and  create)  in  about  half  of  their 
learning  objectives  (see  Table  2).  These  top  three  Bloom’s  levels  are  usually  associated 
with  critical  thinking.  Bloom’s  understand  level  predominates  in  the  other  two  DAU  systems 
engineering  courses. 

Half  of  the  student  learning  objectives  for  the  NPS  SE3100  course  grade  out  at  the 
critical  thinking  level  (e.g.,  analyze,  evaluate,  and  create ;  see  Table  3).  Bloom’s  apply  level 
predominates  in  the  NPS  SI3400  course  and  the  remember  and  understand  levels  are  the 
focus  of  the  NPS  SE4012  course. 

It  should  be  noted  that  Tables  2  and  3  are  not  directly  comparable.  The  assessment 
in  Table  2  is  based  upon  lesson  objectives  in  the  four  DAU  systems  engineering  certification 
courses,  of  which  there  are  a  great  number.  The  assessment  in  Table  3  is  based  upon 
course  objectives  from  the  three  NPS  systems  engineering  courses,  of  which  there  are  but  a 
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few.  Also,  the  number  of  hours  of  CBT,  lecture,  and  problem-based  instruction  in  the  NPS 
courses  was  not  clearly  identified  on  the  syllabi  examined.  Therefore,  no  categorization  of 
instructional  methods  for  the  NPS  courses  was  possible  (and  methods  can  be  expected  to 
vary  by  instructor). 

With  respect  to  course  assessments,  further  research  into  the  types  of  objective  and 
subjective  questions  is  needed  to  conclusively  determine  if  the  questions  really  assess  the 
depth  of  student  learning  (see  Table  4).  However,  deep  learning  by  the  students  taking  the 
SYS  101  and  SYS  201  courses  might  be  encouraged  through  the  use  of  other  assessment 
methods  besides  online  multiple  choice  exams.  While  this  might  seem  difficult  for  computer- 
based  training,  it  is  certainly  not  impossible.  For  example,  computer  simulations,  scored 
games,  and  intelligent  essay  assessors  might  be  used.  The  other  DAU  certification  courses 
and  the  three  NPS  courses  appear  to  have  a  good  mix  of  both  objective  and  subjective 
assessments  that  encourage  a  deep  approach  to  learning. 

Based  upon  this  research,  deep  learning,  which  promotes  the  development  of  critical 
thinking  skills,  can  occur  in  the  SYS  203  and  SYS  302  courses  when  students  are  involved 
in  problem-based  exercises.  However,  both  SYS  101  and  SYS  201  could  be  improved  by 
adding  problem-solving  scenarios  to  stimulate  the  mind  and  help  students  build  more 
sophisticated  mental  models  of  the  systems  engineering  discipline  earlier  in  their  training. 
Also,  more  of  the  lesson  objectives  within  all  of  the  DAU  systems  engineering  courses  could 
be  written  with  verbs  that  target  the  analyze,  evaluate,  and  create  levels  of  Bloom’s 
Taxonomy. 

Some  might  argue  that  the  systems  engineering  fundamentals  course  (SYS  101)  and 
intermediate  course  (SYS  201)  have  to  first  target  the  remember  and  understand  Bloom’s 
levels  before  students  are  able  to  move  on  to  the  analyze,  evaluate,  and  create  levels  that 
develop  student  critical  thinking  skills.  I  would  disagree.  Research  has  shown  that  that  even 
novice  adult  students  benefit  from  learning  approaches  that  build  on  past  experiences. 

From  a  constructivist  point  of  view,  the  goal  of  training  and  education  is  to  develop  within  the 
student  increasingly  sophisticated  ways  of  reasoning  and  problem  solving.  In  effect, 
exposure  to  problem  solving,  even  in  these  initial  courses,  would  help  students  build  bridges 
from  their  current  ways  of  thinking  to  more  correct  contextual  ways  of  thinking  about 
systems  engineering.  Rather  than  filling  their  brains  with  lists  of  terms  and  acronyms  that 
they  may  not  even  be  able  to  apply,  the  goal  should  be  to  correct  any  pre-existing  mental 
models  and  mature  the  right  mental  models  that  students  need  in  order  to  succeed  in  the 
complex  world  of  defense  systems  engineering  (Pratt,  n.d.,  p.  4). 

Similar  observations  can  be  made  regarding  the  three  NPS  systems  engineering 
courses.  Since  the  three  courses  provide  equivalent  credit  to  the  DAU  systems  engineering 
certification  courses,  far-transfer  of  learning  to  the  job  is  essential.  Course  learning 
objectives  requiring  students  to  remember,  understand,  and  apply  could  be  rewritten  to 
challenge  students  to  think  critically  (i.e.,  analyze,  evaluate,  and  create).  Of  course,  this 
would  also  require  that  the  context  for  these  objectives  be  stepped  up  from  lecture  to 
problem-based  instruction  and  that  students  be  assessed  to  the  higher  Bloom’s  levels  of 
functional  thought. 

In  the  balance  of  this  paper,  I  summarize  my  literature  search  that  led  me  to 
conclude  that  deep  learning  and  far-transfer  of  learning  to  the  job  are  best  achieved  using 
problem-based  instruction  that  challenges  students  to  think  critically.  I  also  suggest  areas  for 
further  research. 
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Learning — What  Really  Works? 

Since  1986,  Ken  Bain,  Director  of  the  Center  for  Teaching  Excellence  at  New  York 
University,  has  conducted  ongoing  research  to  identify  and  examine  highly  effective 
university  and  college  teachers.  In  his  book,  What  the  Best  College  Teachers  Do,  one  of 
the  questions  that  Bain  asked  was  the  following:  What  do  the  best  teachers  know  about  how 
students  learn?  Here  are  the  top  four  answers: 

■  Knowledge  is  constructed,  not  received,  and  students  bring  pre-existing 
paradigms  to  the  class  that  shape  how  they  construct  meaning. 

■  Mental  models  change  slowly  and  only  by  challenging  students  intellectually 
(i.e.,  engaging  them  in  deep  thinking). 

■  Questions  are  crucial  because  they  help  students  construct  knowledge. 

■  Caring  is  crucial;  if  students  don’t  ask  important  questions  and  care  about  the 
answers,  they  will  not  try  to  reconcile  or  integrate  new  information  and 
replace  old  mental  models.  (Bain,  1986,  pp.  26-32) 

The  implications  for  how  we  should  teach  systems  engineering  are  significant.  Bain 
reports  that  the  best  teachers  create  a  natural  critical  learning  environment  in  which 
knowledge  and  skills  are  incorporated  into  real-world  (i.e.,  authentic)  tasks  that  engage  the 
student,  arouse  their  curiosity,  and  challenge  their  assumptions  (i.e.,  mental  models).  He 
also  saw  the  best  teachers  create  safe  learning  environments,  where  students  were  free  to 
fail,  receive  feedback,  and  try  again  before  being  assessed.  And  finally,  Bain  found  that  the 
students  understood  and  retained  what  they  had  learned  because  they  had  exercised  their 
reasoning  abilities  to  solve  problems  that  concerned  them  (Bain,  1986,  pp.  46-47).  In  other 
words,  deep  learning  rather  than  surface  learning  had  occurred. 

Bain  (1986)  concludes  the  following: 

The  most  effective  teachers  use  class  time  to  help  their  students  think  about 
information  and  ideas  the  way  scholars  in  the  discipline  do.  They  think  about 
their  own  thinking  and  make  students  explicitly  aware  of  that  process,  constantly 
prodding  them  to  do  the  same.  They  do  not  think  only  in  terms  of  teaching  their 
discipline;  they  think  about  teaching  students  to  understand,  apply,  analyze, 
synthesize,  and  evaluate  evidence  and  conclusions,  (pp.  114-155) 

As  discussed  earlier,  teaching  to  analyze,  evaluate,  and  create  challenges  students  to  think 
critically. 

Learning  Style  Preferences 

One  of  the  potential  traps  with  systems  engineering  instruction  is  falling  back  into 
traditional  methods  of  engineering  instruction.  I  experienced  traditional  instruction  in  my 
undergraduate  years  as  I  pursed  a  Bachelor  of  Science  degree  in  the  engineering  sciences. 
Most  of  my  chemical,  electrical,  mechanical,  and  materials  science  classes  were  taught  as 
lectures.  The  problem  is  that  I’m  a  visual  learner.  I  understand  concepts  and  information 
most  readily  when  they  are  presented  in  pictures  and  flow  charts  or  by  demonstrations. 

Richard  Felder,  a  professor  emeritus  of  chemical  engineering  at  North  Carolina  State 
University  has  studied  the  learning  style  preferences  of  over  2,500  engineering 
undergraduate  students  at  12  universities.  He  and  his  colleagues  have  found  the  following: 

■  82%  of  these  students  are  visual  vice  verbal  learners,  preferring  pictures, 
diagrams,  flow  charts,  and  demonstrations; 
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■  64%  of  these  students  are  active  vice  reflective  learners,  processing 
information  through  engagement  in  physical  activity; 

■  63%  of  these  students  use  their  senses  vice  intuitions,  perceiving  sights, 
sounds,  and  physical  sensations;  and 

■  60%  of  these  students  are  sequential  versus  global  learners,  preferring  a 
logical  progression  of  incremental  steps. 

Yet,  engineering  instruction  at  the  schools  they  attend  is  primarily  verbal,  reflective, 
and  often  intuitive,  emphasizing  theory  and  mathematical  modeling  over  demonstration  or 
the  use  of  visual  aids  (Felder  &  Brent,  2005,  p.  61).  Could  this  mismatch  of  learning  style 
preference  and  methods  of  instruction  be  a  problem  in  systems  engineering  training  and 
education  within  the  Department  of  Defense?  Perhaps  future  research  could  sample  the 
learning  style  preferences  of  systems  engineering  students  taking  the  four  DAU  certification 
courses  (and  equivalent  courses)  and  compare  those  student  preferences  with  the  teaching 
styles  of  the  instructors. 

We  have  intuitively  known  that  a  picture  is  worth  a  thousand  words.  When  compared 
with  written  words  or  verbal  communications,  people  actually  do  communicate  more  simply 
and  efficiently  with  pictures  or  visual  images.  This  is  due  to  our  natural  ability  to  process 
and  retain  visual  images  in  our  minds.  Pictures  are  information-rich  and  can  convey  more 
precise  meanings  and  more  clearly  depict  ideas  (Gerard  &  Goldstein,  2005,  pp.  18,  45). 
Learning  transfer  can  also  be  improved  with  images.  With  images,  patterns  emerge, 
revealing  relationship.  These  patterns  also  help  in  understanding  how  processes  work. 
Communicating  ideas  with  a  visual  image  can  result  in  clearer  understanding  of  complex 
processes  (see  also  Mintzberg  &  Westley,  2001,  pp.  92-93). 

Kevin  Forsberg,  Hal  Mooz,  and  Howard  Cotterman  (2005)  from  the  Center  for 
Systems  Management  have  dedicated  the  third  edition  of  Visualizing  Project  Management 
to  “mastering  complexity”  (p.  xxi).  They  say  that  logical  and  systematic  project  management 
and  systems  engineering  processes  are  left-brain  activities.  To  stimulate  creativity,  the 
visually  oriented  right-brain  needs  to  be  engaged.  Therefore,  their  book  is  full  of  visual 
models  that  simplify  these  complex  process  and  help  the  student  understand  how  things 
really  work  (Forsberg,  Mooz,  &  Cotterman,  2005,  pp.  xxiv-xxv).  In  particular,  they  use  the 
“V”  model  to  depict  the  systems  engineering  process  of  top  down  requirements 
decomposition  and  design  definition  and  bottom-up  system  integration  and  validation.  It 
should  be  noted  that  the  DAU  and  NPS  systems  engineering  courses  studied  in  this 
research  all  make  good  use  of  visual  models. 

Far-Transfer  of  Learning  to  the  Job 

In  her  book,  Building  Expertise:  Cognitive  Methods  for  Training  and  Performance 
Improvement,  Ruth  Colvin  Clark  (2008)  discusses  the  psychology  of  learning  transfer  and 
practical  ways  to  teach  for  transfer.  She  posits  that  far-transfer  of  learning,  the  ability  to 
solve  ill-defined  or  ambiguous  problems  on  the  job,  comes  from  creative  and  critical  thinking 
(Clark,  2008,  pp.  234,  245).  Yet,  far-transfer  of  learning  does  not  result  from  a  single 
training  event.  In  addition  to  training,  far-transfer  requires  an  innovative  culture, 
collaborative  projects,  diverse  work  experiences,  and  the  ability  to  reason  within  unfamiliar 
contexts  or  on  novel  tasks  (i.e.,  fluid  intelligence;  Clark,  2008,  p.  249). 

To  promote  far-transfer  of  learning,  Clark  recommends  the  inductive  training 
technique.  Inductive  training  can  be  described  by  comparing  it  with  traditional  training. 
During  traditional  training,  the  instructor  presents  the  content,  the  instructor  provides 
examples,  and  the  students  apply  the  content.  Inductive  training  changes  the  sequence  and 
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puts  more  emphasis  on  active  engagement  of  the  students:  the  instructor  provides 
examples,  the  students  derive  the  content,  and  the  students  apply  the  content.  Traditional 
training  actively  engages  the  students  only  one  third  of  the  time;  inductive  training  actively 
engages  the  students  two  thirds  of  the  time  (Clark,  2008,  p.  270). 

According  to  Clark  (2008),  the  reason  that  inductive  learning  enables  far-transfer  of 
learning  is  because  the  students  are  engaged  in  building  a  personal  mental  model  based  on 
their  own  experience  and  collaboration  with  other  students.  Clark  also  recommends  the  use 
of  simulations  (used  in  a  guided  discovery  mode)  and  problem-centered  instruction.  Both 
methods  promote  far-transfer  of  thinking  skills  by  engaging  the  students  to  build  their  own 
knowledge  and  skill  base  in  long-term  memory  (i.e.,  mental  models)  within  a  real-world 
context  (Clark,  2008,  pp.  273,  283-285). 

Many  others  agree  with  Clark.  Nobel  laureate  Herbert  A.  Simon,  a  professor  of 
psychology  and  computer  science  at  Carnegie  Mellon  University  who  studies  human 
decision  making  has  concluded  that  experience  (e.g.,  from  a  problem-solving  exercise) 
enables  us  to  “chunk”  information  so  that  we  can  store  and  retrieve  it  more  easily  (as 
reported  by  Hayashi,  2001 ,  p.  7).  Felder  and  Brent  (2005)  say  that  inductive  teaching 
methods  such  as  problem-based  and  project-based  learning  can  motivate  students  by 
making  subject  matter  relevant  to  prior  and  future  experiences,  emphasizing 
conceptualization,  versus  rote  memorization  (p.  64). 

Learning  for  Rapid  Cognition 

In  his  book  Blink:  The  Power  of  Thinking  without  Thinking,  Malcolm  Gladwell  (2005) 
explains  how  rapid  cognition  that  happens  in  a  blink  of  an  eye  can  be  used  to  make  fairly 
good  decisions  in  otherwise  complex  situations.  Psychologists  call  the  critical  part  of  rapid 
cognition  thin-slicing,  which  refers  to  the  ability  of  our  subconscious  mind  to  recognize 
patterns  in  everyday  life  situations  based  upon  narrow  slices  or  samples  of  experience 
(Gladwell,  2005,  p.  23).  For  example,  I  can  tell  by  my  wife’s  voice,  within  the  blink  of  an  eye, 
if  she  is  happy,  sad,  or  mad.  Even  on  the  telephone,  the  patterns  of  her  voice — just  her  first 
few  words — give  me  all  the  clues  I  need  to  correctly  determine  her  mood.  This  is  based 
upon  my  experience  in  listening  to  her  and  the  fact  that  I  love  her  dearly.  I  have  created  in 
my  mind  an  array  of  mental  models  of  her  different  voice  patterns.  Can  rapid  cognition  be 
useful  in  training  systems  engineers  to  recognize  and  act  on  problems  even  in  the  complex 
environment  of  defense  systems  engineering?  Perhaps  it  can. 

Gladwell  (2005)  tells  the  story  of  Cook  County  Hospital  (Chicago,  IL)  that  has  a 
trauma  center  that  inspired  the  television  series  ER.  Faced  with  overwhelming  costs  and  a 
shoe-sting  budget  Brendan  Reilly,  chairman  of  the  hospital’s  Department  of  Medicine, 
turned  to  cardiologist  Lee  Goldman  who,  based  upon  his  years  of  experience  with  heart 
attacks,  came  up  with  an  equation  for  predicting  if  chest  pains  really  meant  that  a  heart 
attack  was  about  to  happen.  In  the  past,  doctors  would  ask  lots  of  questions  of  the  patient, 
ask  for  expensive  tests,  and  as  a  precaution,  admit  the  patient.  When  Goldman’s  decision 
tree  (i.e.,  pattern  analysis)  was  implemented  in  the  hospital  emergency  room  over  a  two- 
year  period,  diagnoses  were  70%  better  than  the  old  method.  The  point,  according  to 
Gladwell,  is  that  too  much  information  confuses  the  issue  and  makes  it  harder  to  pick  up  the 
basic  signature  of  the  problem  (i.e.,  the  pattern;  Gladwell,  2005,  pp.  125-136,  142). 

Nobel  laureate  Herbert  Simon  has  concluded  that  “experts  see  patterns  that  elicit 
from  memory  the  things  they  know  about  such  situations  [and]... what  distinguishes  experts 
is  that  they  have  very  good  encyclopedias  that  are  indexed  and  pattern  recognition  that  is 
that  index”  (as  quoted  in  Hayashi,  2001,  p.  63).  So,  what  patterns  should  we  be  teaching 
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our  systems  engineering  students?  For  example,  are  there  patterns  in  technical  reviews, 
earned  value  analysis,  or  risk  assessments  that  could  instantaneously  (in  the  blink  of  an 
eye)  let  them  know  whether  a  problem  exists?  Moreover,  if  every  systems  engineer  working 
for  or  with  the  Department  of  Defense  used  decision  trees  prepared  by  systems  engineering 
experts,  would  our  system  acquisition  programs  have  better  outcomes?  Twenty  years  ago 
when  I  attended  the  Defense  Systems  Management  College  Program  Management  Course, 
I  recall  the  recommendation  to  use  the  Willoughby  templates  to  identify  risk  areas  when 
transferring  systems  from  development  into  production.  Today  these  templates  have  been 
incorporated  into  the  Best  Manufacturing  Practices  Center  of  Excellence  (BMPCOE) 
Technology  Risk  Identification  and  Management  Systems  (TRIMS).  Would  more  emphasis 
on  the  use  of  such  expert  templates  simplify  issue  and  risk  identification  for  earlier 
responses  and  ultimately  help  our  acquisition  programs  succeed? 

Learning  Patterns  of  Response 

According  to  UCLA  Professor  Moshe  Rubinstein,  an  internationally  renowned 
authority  on  problem  solving  and  creativity  in  organizations,  “We  must  learn  to  live 
harmoniously  with  change,  chaos,  and  uncertainty.  It  is  now  the  age  of  the  brain.  It  is  the 
age  of  finding  ways  to  tap  more  of  the  human  potential  for  creativity  and  innovation,  to  learn 
to  adapt  to  chaos  and  uncertainty,  and  to  use  our  minds  to  establish  a  sense  of  purpose  and 
meaning  in  our  personal  and  professional  lives”  (Rubinstein  &  Firstenberg,  1999,  p.  20).  In 
an  age  of  growing  connectivity  and  complexity,  to  include  more  complex  defense  systems, 
we  must  be  able  to  embrace  uncertainty,  change,  and  chaos.  According  to  Rubinstein,  the 
human  brain  has  the  capacity  to  do  just  that. 

During  a  recent  weekend  getaway  to  Marin  County,  CA,  my  bride  and  I  took  some 
time  to  visit  the  national  office  and  kennels  of  Guide  Dogs  for  the  Blind  in  San  Rafael.  While 
touring  the  kennels,  we  were  told  that  the  young  Labrador  Retrievers  are  actually  trained  to 
respond  in  patterns  of  behavior.  Clearly,  the  training  course  can  never  simulate  all  of  the 
possible  obstacles  (to  include  change,  chaos  and  uncertainty)  in  a  city,  home,  work,  or 
recreational  environment  that  these  young  Labs  will  encounter.  So,  the  Labs  are  taught 
“patterns  of  response”  in  order  to  lead  the  blind  person  around  obstacles  in  their  path.  Can 
systems  engineering  training  and  education  take  a  lesson  from  how  guide  dogs  are  trained? 

Robert  C.  Collins,  MD,  a  professor  and  the  chair  of  the  Department  of  Neurology  at 
the  UCLA  School  of  Medicine,  has  discovered  that  brain  wave  patterns  for  hand  movements 
are  unique,  but  there  is  about  a  50%  overlap  across  various  patterns.  This  means  that  hand 
movements  start  out  planned,  but  end  up  as  unplanned  responses  to  the  environment 
(Rubinstein  &  Firstenberg,  1999,  p.  49).  For  example,  I  play  the  slide  trombone.  Let’s 
assume  that  I’m  going  to  try  to  play  a  solo  and  have  never  seen  the  music  before  (I’m  sight 
reading).  Based  on  past  experience,  my  brain  knows  how  far  to  extend  my  arm  and  wrist  to 
reach  the  slide  to  the  4th  position  G  to  start  my  solo  (i.e.,  the  planned  brain  waves).  But,  the 
next  note  is  B-flat.  Do  I  play  that  note  in  the  first  position  or  the  fifth  position?  Either 
position  will  work.  And,  if  I  encounter  eighth  or  sixteenth  notes,  the  next  series  of  notes  up 
or  down  the  slide  could  happen  in  a  blink  of  the  eye.  How  then  does  my  brain  know  what  to 
do  next  (i.e.,  handle  the  unplanned)?  Answer:  spontaneous  improvisation  from  previous 
experience  (patterns  of  response).  Even  though  I’ve  never  seen  the  music  before,  I’ve 
stored  patterns  of  rules  in  my  brain  for  getting  to  the  next  note(s)  quickly  and  efficiently.  In 
the  case  of  the  G  to  B-flat,  I’ll  look  to  the  note(s)  after  the  B-flat  to  decide  if  it  is  easier  to  use 
the  first  or  fifth  position,  thereby  being  better  prepared  to  play  the  subsequent  notes. 
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Rubinstein  says,  “We  can  safely  conclude  that  human  experience  almost  always 
involves  both  the  earlier  stored  part,  which  is  reproduced,  and  the  newly  created  part,  which 
is  produced”  (Rubinstein  &  Firstenberg,  1999,  p.  49).  How  might  this  knowledge  of  our  how 
the  brain  works  and  stored  rule/response  patterns  change  the  way  in  we  teach  systems 
engineering?  Might  we  teach  patterns  of  response  that  could  ultimately  be  applied  to  solve 
complex  systems  engineering  issues  and  mitigate  risks? 

Rubinstein  also  has  an  interesting  perspective  on  creative  thinking.  “Creative 
thinking  requires  a  process  that  is  quite  different  from  that  of  rational  thinking.  Whereas 
rational  thinking  depends  on  categories  and  labels  that  have  been  set  up  in  advance, 
creative  thinking  demands  that  we  form  new  categories  and  labels.  Rational  thought  leads 
us  to  find  the  similarities  between  a  new  experience  and  previous  experiences  so  that  we 
can  treat  them  the  same  way.  Creative  thought  looks  for  the  differences  among 
experiences,  seeking  unique  ways  of  both  interpreting  situations  and  acting  upon  them. 
Rational  thinking  seeks  to  confirm;  creative  thinking  seeks  to  invent.”  (Rubinstein  & 
Firstenberg,  1999,  p.  22).  Perhaps  this  definition  of  creative  thinking  should  also  be  used  to 
guide  and  assess  the  success  of  our  systems  engineering  problem-based  exercises. 

Conclusions 

Knowledge,  skills,  and  abilities  within  the  discipline  of  systems  engineering  are  best 
learned  experientially  through  problem-based  instruction.  Opportunities  to  role  play, 
simulate,  or  actually  perform  system  engineering  tasks  really  help  the  students  transfer 
learning  from  the  classroom  to  their  work.  Over  the  years,  the  most  successful  training  and 
education  programs  I’ve  participated  in  as  a  member  of  the  defense  acquisition  workforce 
have  been  case  studies  and  simulations  that  combine  the  technical  aspects  of  the  systems 
engineering  discipline  with  activities  that  require  the  application  of  interpersonal  skills  and 
leadership.  Having  been  an  instructor  in  both  systems  engineering  training  and  education 
environments,  I  know  that  students  do  their  best  when  challenged  with  authentic  problems 
that  have  meaning  to  them  in  the  real-world.  As  adult  students,  they  appreciate  a  learning 
environment  in  which  they  can  “do  it  until  they  get  it  right.”  Knowingly  or  unknowingly,  they 
can  learn  much  from  their  peers.  Also,  they  excel  when  invited  to  display  their  knowledge  in 
front  of  their  peers.  All  of  the  experience  I’ve  had  in  learning  and  teaching  within  this 
discipline  point  to  the  absolute  necessity  for  active  learning  activities  that  are  relevant  to  the 
real-world  of  the  systems  engineer. 

The  purpose  of  instruction  in  the  defense  systems  engineering  discipline  is  to  equip 
adult  students  to  succeed  in  what  can  be  a  very  complex  and  often  ambiguous  public  policy 
environment.  These  students  want  to  understand  the  “why”  behind  the  concepts  and 
principles  of  their  profession.  Only  with  that  knowledge  can  they  know  what  is  important  and 
what  to  ignore  when  overloaded  with  information.  Also,  they  need  to  have  had  opportunities 
in  a  nonthreatening  academic  environment  to  experience  what  happens  when  they  ignore 
that  which  is  important  (i.e.,  learn  from  their  mistakes).  Moreover,  they  need  to  think  deeply 
within  the  discipline  to  understand  what  to  accept  and  what  to  challenge.  In  other  words 
they  need  to  be  humble  critics  of  their  profession  who  can  rationally  argue  for  change  when 
change  is  needed.  As  an  instructor,  I  need  to  come  alongside  my  students  (current  and 
future  systems  engineers)  to  awaken  and  develop  their  intellects  in  the  following  key  areas: 

■  Intellectual  Humility — the  systems  engineering  discipline  is  so  big  and 
dynamic  that  no  one  person  can  ever  know  everything. 
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■  Intellectual  Empathy — the  systems  engineer  must  be  able  to  understand  the 
perspectives  and  objectives  of  all  acquisition  stakeholders  (e.g., 
warfighters/users,  Congress,  Executive  branch,  and  defense  industry). 

■  Intellectual  Autonomy — in  defending  the  program’s  systems  engineering 
approach,  the  systems  engineer  has  to  be  able  to  justify  why  he/she  tailored 
a  systems  engineering  process  model,  technical  reviews,  audits,  verifications, 
etc. 

■  Intellectual  Integrity — responsibility  and  accountability  for  program  goals,  to 
include  credible  cost,  schedule,  and  performance  reporting,  are  required  from 
the  systems  engineer. 

■  Confidence  in  Reason — the  systems  engineer  must  develop  sound  rationale 
for  the  development  approach,  testing  strategies,  and  logistical  support  for 
the  system. 

■  Fair-mindedness — the  systems  engineer  is  a  public  servant,  expected  to  give 
due  consideration  to  all  viewpoints  and  avoid  even  the  appearance  of  any 
conflict  with  his/her  personal  interests  or  ambitions. 

These  intellectual  traits  (“Foundation  for  Critical  Thinking,”  1996)  can  only  be 
awakened  and  developed  through  a  deep  approach  to  learning.  My  students  need  to  be 
challenged  to  think  beyond  the  course  and  look  to  the  expert  application  of  the  knowledge 
they  are  learning  as  it  affects  their  real-world  jobs.  To  do  this,  I  need  to  provide  learning 
activities  that  target  Bloom’s  evaluate  level  and  frequently  go  above  that  to  the  create  level. 

I  have  to  prepare  learning  objectives  and  assessments  that  go  beyond  simply  remembering 
facts  and  applying  procedures.  I  need  to  create  a  critical  natural  learning  environment  that 
invites  students  to  test  the  boundaries  of  the  discipline  (Bain,  2004,  p.  99).  A  learning 
environment  that  invites  my  students  to  argue,  compare,  rate  and  ultimately  judge  for 
themselves  what  works  and  what  doesn’t  work  is  what  I'm  seeking.  After  learning  activities, 

I  need  to  give  the  students  time  to  reflect  deeply  on  what  they  have  experienced.  In  so 
doing,  I  want  them  to  see  the  patterns  of  thought  that  led  them  to  their  conclusions.  By 
recognizing  these  patterns,  they  can  begin  to  experience  the  power  of  thinking  without 
thinking  -  like  the  experts  do  (i.e.,  rapid  cognition  based  on  thin-slicing,  per  Gladwell, 

2005). 

Critical  natural  learning  environments  can  be  cultivated  and  observed  through 
classroom  and  online  discussions  of  real-world  case  studies.  Such  environments  can  also 
be  achieved  through  an  integrated  course  exercise  where  student-led  teams  develop  and 
brief  a  technical  systems  engineering  strategy  pertaining  to  a  real-world  need.  Role  playing 
during  classroom  discussions  of  dilemmas  faced  in  real-world  case  studies  would  also  work 
nicely.  However,  the  one  intangible  in  all  of  these  learning  activities  is  my  passion  and 
motivation  for  learning  the  systems  engineering  discipline.  It  is  the  creativity  and  drive  that  I 
bring  to  the  course  and  into  the  classroom  that  truly  motivates  my  students.  To  keep  the 
energy  and  motivation  flowing,  I  must  constantly  improve  the  learning  activities  and  keep 
them  relevant,  gain  a  better  understanding  of  where  my  students  are  coming  from 
experientially  and  professionally,  and  be  responsive  to  the  constructive  feedback  my 
students  give  me. 
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Overview 


Literature  Search:  How  do  today's  educators  define 
effective  training  and  education? 

Research  Question:  Do  DoD's  Systems  Engineering 
courses  encourage  a  deep  approach  to  learning  and 
the  far-transfer  of  learning  to  the  job ? 

Approach:  Examine  DAU  and  NPS  Systems  Engineering 
courses: 

-  Learning  Outcomes  (Objectives) 

-  ivietnoas  of  instruction 

-  Learning  Assessments  2 


Learning  -  What  Really  Works? 


•  Knowledge  is  constructed,  not  received 

•  Mental  models  change  slowly  and  only 
by  challenging  students  intellectually 


*  Questions  are  crucial  because  they 
help  students  construct  knowledge 


•  Caring  is  crucial;  if  students  don't  ask  important 
questions  and  care  about  the  answers,  they  will  not  try 
to  reconcile  or  integrate  new  information  and  replace 
old  mental  models 

Source:  Ken  Bain,  What  the  Best  College  Teachers  Do 


Far-Transfer  of  Learning  to  the  Job 

•  Requires  creative  and  critical  thinking, 

•  stimulated  by  inductive  training,  and 

•  problem-centered  instruction 

•  that  engages  students  to  build  their  own 
knowledge  and  skill  base  in  long-term  memory 
within  a  real-world  context 

Source:  Ruth  Colvin  Clark,  Building  Expertise 


DAU  Systems  Engineering  Course  Hours 
Categorized  by  Method  of  Instruction 


■  Problem-Based  Hours 

■  Lecture  Hours 

■  CBT  Hours 


SYS  101  SYS  202  SYS  203  SYS  302  Totals 


SE  203  Tower  Exercise 


Hrooiem-Lenterea 

Instruction 


Challenging  the  Students' 
Mental  Models 
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Teaching  Around  the  Cycle 

(Learning  Style  Preferences) 


Source:  R.  Felder  &  R.  Brent,  Understanding  Student  Differences 


Engineering  Student  Learning  Style  Preferences 
vs.  DoD  SE  Courses  Instructional  Methods 


Sensory 


Global 


Ketlective 


Verbal 


Visual 


Active 


Sequential 


Intuitive 


— Engineering  Student 

Learning  Style  Preferences 
(Felder  &  Brent,  2005) 

— DoD  SE  Courses 

Instructional  Methods 
(Fast,  2011) 
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Bloom's  Taxonomy 


Analysis 


Analyzing 


Application 


Applying 


omprehension 


Knowledge 


Critical 

Thinking 


Creating 


old  version 


new  version 


DAU  Systems  Engineering  Lesson  Objectives 
Categorized  by  Bloom's  Level 


Critical 

Thinking 


■  Create 

■  tvaiuate 

■  Analyze 

■  Apply 

■  Understand 

■  Remember 


SYS  101  SYS  202  SYS  203  SYS  302  Totals 
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NPS  Systems  Engineering  Course  Objectives 
Categorized  by  Bloom's  Level 


SE  3100 


SI  3400 


SE  4012 


Totals 


li 


Types  of  Learning  Assessments  Used  In 
DAU  and  NPS  Systems  Engineering  Courses 


Assessment 

SYS  101 

SYS  202 

SYS  203 

SYS  302 

SE  3100 

SI  3400 

SE  4012 

Objective  Exam/Quiz 

X 

X 

X 

X 

X 

X 

X 

Subjective  Exam/Quiz 

X 

X 

X 

Homework 

X 

X 

X 

Discussion  Participation 

X 

X 

X 

X 

Reflective  Writing 

X 

Individual  Briefing 

X 

X 

X 

X 

X 

Individual  Project 

X 

X 

X 

Team  Project 

X 

X 

X 

X 

X 
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Recommendations 

•  SYS  101  and  SYS  201  could  be  improved  by  adding  problem 
solving  scenarios  to  stimulate  the  mind  and  help  students 
build  more  sophisticated  mental  models  of  the  systems 
engineering  discipline  earlier  in  their  training 

•  To  promote  critical  thinking,  more  of  the  lesson  objectives 
within  all  of  the  DAU  and  NPS  systems  engineering  courses 
could  be  written  with  verbs  that  target  the  analyze,  evaluate, 
and  create  levels  of  Bloom's  Taxonomy 

•  Additional  research  into  learning  for  rapid  cognition  and 
learning  patterns  of  response  might  reveal  other  ways  to 
improve  the  effectiveness  of  systems  engineering  training  and 
education 
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Learning  for  Rapid  Cognition 

...the  ability  of  our  subconscious  mind  to 
recognize  patterns  in  ever  da  life 
situations  based  upon  narrow  slices  or 
samples  of  experience... 

Source:  Malcolm  Gladwell,  Blink: 

The  Power  of  Thinking  Without  Thinking 


"Experts  see  patterns  that  elicit  from  memory  the  things 
they  know  about  such  situations  ^and^...what  distinguishes 
experts  is  that  they  have  very  good  encyclopedias  that  are 
indexed  and  pattern  recognition  that  is  that  index" 

Source:  Nobel  laureate  Herbert  Simon 
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Learning  Patterns  of  Response 


•  The  human  brain  has  the  ability  to  adapt  to 
chaos,  uncertainty,  and  change 

•  Human  experience  consists  of  about: 

50  percent  stored  responses,  and 
50  percent  newly  created  responses 

•  Can  we  teach  more  "patterns  of  response" 
to  help  systems  engineers  better  solve 
complex  acquisition  problems? 


Ahvih*  I:  KnhtiiiviL-iii  . i ; i >i  3m  K  Hnmbeffl 
L  InLU-raijf  iif'  i  jJiliirnu,  L  r 1  ■.  .injylfL 


BUSINESS 


into 

Solutions 


Source:  Moshe  Rubinstein  &  Iris  Firstenberg,  The  Minding  Organization 
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